ImpactMojo ImpactMojo
Premium

Multivariate Analysis 101

Advanced Methods, Diagnostics & Applied Problem Sets
ImpactMojo Workshop Series • Hands-On Policy Analysis Practice
75-90 Minutes

Workshop 2: Interactions, Diagnostics & Real-World Applications

Target Audience: Researchers and policy analysts ready to apply advanced multivariate techniques to complex datasets

Prerequisites: Workshop 1 or solid foundation in basic multiple regression

Materials Needed: Statistical software with diagnostic capabilities, real policy datasets, calculators

Learning Objectives

By the end of this workshop, participants will be able to:

Problem Set 1: Interaction Effects in Education Policy

Intermediate
Dataset: School Performance in Rural Maharashtra

Research Question: Does the effect of teacher training on student learning outcomes differ by school infrastructure quality?

Variables:

  • test_score: Average class 5 mathematics score (0-100)
  • teacher_training: Binary (1 = teacher received training, 0 = no training)
  • infrastructure: Infrastructure index (0-10 scale)
  • student_ses: Average student socioeconomic status (-2 to +2)
  • class_size: Number of students in class
  • years_experience: Teacher's years of experience

Problem 1.1: Interaction Model Specification

Your task: Set up a regression model to test whether infrastructure moderates the teacher training effect.

Model Setup Exercise

Step 1: Write the full regression equation including the interaction term:

test_score = β₀ + β₁(teacher_training) + β₂(infrastructure) + β₃(_____) + β₄(student_ses) + β₅(class_size) + β₆(years_experience) + ε

Step 2: What does each coefficient represent?

  • β₁: Effect of teacher training when infrastructure = _____ and other variables held constant
  • β₂: Effect of infrastructure when teacher_training = _____ and other variables held constant
  • β₃: How much the effect of _____ changes per unit increase in _____
# R Code for Interaction Model model_interaction <- lm(test_score ~ teacher_training + infrastructure + teacher_training * infrastructure + student_ses + class_size + years_experience, data = school_data) # Stata Code regress test_score teacher_training infrastructure /// c.teacher_training#c.infrastructure student_ses class_size years_experience # Python Code import statsmodels.formula.api as smf model = smf.ols('test_score ~ teacher_training + infrastructure + teacher_training:infrastructure + student_ses + class_size + years_experience', data=school_data).fit()

Problem 1.2: Interaction Interpretation Practice

Given these hypothetical results:

Variable Coefficient Std Error p-value
teacher_training 8.5 2.1 0.001
infrastructure 3.2 0.8 0.001
teacher_training × infrastructure 1.7 0.6 0.006
student_ses 12.3 1.9 <0.001
Interpretation Questions
  1. Schools with poor infrastructure (infrastructure = 2):
    Effect of teacher training = 8.5 + (1.7 × 2) = _____ points
  2. Schools with good infrastructure (infrastructure = 8):
    Effect of teacher training = 8.5 + (1.7 × _____) = _____ points
  3. Policy insight: Teacher training is (more/less) effective in schools with better infrastructure by _____ points per unit of infrastructure improvement.
  4. Practical significance: The difference in training effectiveness between poor (infrastructure=2) and excellent (infrastructure=10) schools is _____ points.

Problem Set 2: Regression Diagnostics Workshop

Advanced
Diagnostic Challenge: MGNREGA Wage Analysis

Context: You're analyzing factors affecting daily wages in MGNREGA across 500 districts. After running your initial model, you need to check key assumptions.

Initial Model: daily_wage = β₀ + β₁(unemployment_rate) + β₂(literacy_rate) + β₃(distance_to_city) + β₄(state_budget_pc) + ε

Linearity Check

Test: Residuals vs. Fitted plot

Look for: Random scatter around zero

Problem signs: Curved patterns, funneling

Solutions: Log transformation, polynomial terms

Normality of Residuals

Test: Q-Q plot, Shapiro-Wilk test

Look for: Points along diagonal line

Problem signs: Systematic deviations

Solutions: Robust standard errors, transformations

Homoskedasticity

Test: Breusch-Pagan test, residuals vs. fitted

Look for: Constant variance across fitted values

Problem signs: Funneling pattern

Solutions: Robust SEs, weighted least squares

Multicollinearity

Test: VIF (Variance Inflation Factor)

Look for: VIF < 5 (ideally < 3)

Problem signs: VIF > 10

Solutions: Drop variables, ridge regression

Hands-On Diagnostic Exercise (20 minutes)

Step 1: VIF Interpretation (5 minutes)

Given VIF values:

  • unemployment_rate: VIF = 2.1
  • literacy_rate: VIF = 8.7
  • distance_to_city: VIF = 1.8
  • state_budget_pc: VIF = 9.2

Analysis:

  1. Which variables show concerning multicollinearity? _____
  2. What percentage of literacy_rate's variance is explained by other predictors? _____% (Hint: 1 - 1/VIF)
  3. Should you drop any variables? Which ones and why?
Step 2: Heteroskedasticity Test (5 minutes)

Breusch-Pagan test results:

  • Test statistic: 18.7
  • p-value: 0.003
  • Null hypothesis: Homoskedasticity (constant variance)

Questions:

  1. Do you reject the null hypothesis? _____ (Yes/No)
  2. What does this mean for your model? _____
  3. What are two appropriate solutions?
Step 3: Outlier Detection (10 minutes)

Identify problematic observations:

District Standardized Residual Cook's Distance Leverage
District A -3.2 0.15 0.08
District B 1.8 0.003 0.12
District C 2.1 0.25 0.15

Thresholds: |Standardized residual| > 2.5, Cook's D > 4/n, Leverage > 2p/n

Analysis:

  1. Which district is an outlier? _____
  2. Which has high influence? _____
  3. Should any observations be investigated further? Why?

Problem Set 3: Model Selection and Comparison

Advanced
Policy Research Scenario

Context: You're evaluating the effectiveness of digital literacy programs on women's economic empowerment across 300 villages in rural India. You have multiple competing models and need to select the best one.

Model 1: Basic Model

R² = 0.34, AIC = 2847

empowerment_score ~ digital_program + age + education + household_income

Model 2: With Interactions

R² = 0.41, AIC = 2801

empowerment_score ~ digital_program + age + education + household_income + digital_program:education

Model 3: Full Model

R² = 0.45, AIC = 2823

empowerment_score ~ digital_program + age + education + household_income + digital_program:education + village_infrastructure + social_capital

Problem 3.1: Model Selection Criteria

Selection Analysis
  1. R² criterion: Which model explains the most variance? Model _____ with _____% variance explained.
  2. AIC criterion: Which model has the best AIC (lowest value)? Model _____ with AIC = _____.
  3. Parsimony principle: Model 2 vs Model 3 - the improvement in R² is _____ percentage points, but AIC gets (better/worse) by _____ points.
  4. Your recommendation: Which model would you choose and why?

Model Selection Guidelines:

  • AIC difference < 2: Models are essentially equivalent
  • AIC difference 2-10: Some evidence for lower AIC model
  • AIC difference > 10: Strong evidence for lower AIC model
  • R² vs AIC: AIC penalizes complexity, R² doesn't - prefer AIC for model selection

Problem 3.2: Policy Interpretation with Interactions

Selected Model 2 results:

Variable Coefficient Std Error p-value
digital_program 2.8 1.2 0.020
education (years) 0.6 0.15 <0.001
digital_program × education 0.4 0.18 0.035
Policy Memo Calculations
  1. For women with no formal education (education = 0):
    Digital program effect = 2.8 + (0.4 × 0) = _____ points
  2. For women with 8 years education:
    Digital program effect = 2.8 + (0.4 × _____) = _____ points
  3. For women with 12 years education:
    Digital program effect = _____ + (_____ × 12) = _____ points
  4. Policy insight: Each additional year of education increases the program's effectiveness by _____ points.
Policy Implications
  1. Should digital literacy programs target women with higher education levels? Why or why not?
  2. How might you design interventions to help women with lower education benefit more from digital programs?
  3. What additional variables would strengthen this analysis?
  4. How would you communicate these findings to policymakers who aren't familiar with interaction terms?

Problem Set 4: Non-Linear Relationships

Intermediate

Problem 4.1: Polynomial Terms in Income Analysis

Research Question: Does the relationship between experience and income follow a quadratic pattern (increasing returns early in career, diminishing returns later)?

Quadratic Model Specification

monthly_income = β₀ + β₁(experience) + β₂(experience²) + β₃(education) + β₄(urban) + ε

Key insight: The turning point occurs at experience = -β₁/(2×β₂) years

Given Results
  • β₁ (experience) = 2,400
  • β₂ (experience²) = -45
  • β₃ (education) = 1,200

Calculations:

  1. Turning point: Peak income occurs at _____ years of experience
  2. At 10 years experience: Marginal effect = 2,400 + 2(-45)(10) = _____ rupees per additional year
  3. At 30 years experience: Marginal effect = 2,400 + 2(-45)(30) = _____ rupees per additional year
  4. Interpretation: After _____ years, additional experience actually decreases income.

Workshop Synthesis

Advanced multivariate analysis is about telling nuanced stories with data. Interactions reveal that "one size fits all" policies often don't work. Diagnostics ensure your statistical story is credible. Model selection balances complexity with parsimony. Always ask: "What is the policy-relevant insight, and how confident should we be in it?"

Advanced Resources & Software

Diagnostic Tools:

Advanced Econometric Methods:

Policy Application Examples:

Next Steps in ImpactMojo:

This handout is part of the ImpactMojo 101 Knowledge Series
Licensed under CC BY-NC-ND 4.0 • Free to use with attribution • www.impactmojo.in

For complete problem set solutions, diagnostic code scripts, and additional practice datasets, visit the ImpactMojo platform.